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Abstract 

While online instructors may be versed in conducting e-Research (Hung, 2012; Thelwall, 2009), today's 
faculty are probably less familiarized with the rapidly advancing fields of bibliometrics and informetrics. 
One key feature of research in these areas is Citation Analysis, a rather intricate operational feature 
available in modern indexes such as Web of Science, Scopus, Google Scholar, and PsycINFO. This paper 
reviews the recent extant research on bibliometrics within the context of citation analysis. Particular 
focus is on empirical studies, review essays, and critical commentaries on citation-based metrics across 
interdisciplinary academic areas. Research that relates to the interface between citation analysis and 
applications in higher education is discussed. Some of the attributes and limitations of citation 
operations of contemporary databases that offer citation searching or cited reference data are 
presented. This review concludes that: a) citation-based results can vary largely and contingent on 
academic discipline or specialty area, b) databases, that offer citation options, rely on idiosyncratic 
methods, coverage, and transparency of functions, c) despite initial concerns, research from open access 
journals is being cited in traditional periodicals, and d) the field of bibliometrics is rather perplex with 
regard to functionality and research is advancing at an exponential pace. Based on these findings, online 
instructors would be well served to stay abreast of developments in the field. 

Keywords: Bibliometrics, informetrics, citation analysis, information technology, Open resource and 
electronic journals 

INTRODUCTION 

In an ever increasing manner, the educational field is irreparably linked to advances in information 
technology (Plomp, 2013). Moreover, due to the information explosion and exponential growth of 
scholarly knowledge, both students in higher education and faculty face an increasing educational 
challenge to keep pace with developments in their area of study (Varshney, 2012). At the same time, 
online information repositories such as open access archives, conference proceedings, and Web pages 
are facilitating the seismic rate of growth in scientific literature (Larsen & von Ins, 2010). The 
connectivity and promulgation of the knowledge base of scholarly literature serves as a barometer of 
the health of a discipline (Poor, 2009). To add to the complexity, particularly in recent years, cross¬ 
fertilization of academic disciplines and resultant research has significantly increased the challenge of 
evaluating the veracity, robustness, and generalizability of scholarship (Ruscio, Seaman, D'Oriano, 
Stremlo, & Mahalchik, 2012). Conversely, the majority of contemporary scholarly literature is highly 
specialized and, thus, evaluating the merits of research studies has become an onerous and perplexing 
task. This dilemma has been keenly noted by Porter (2012), "Should citations of review essays be 
credited to the same extent as references to original research? Are theoretical papers cited in similar 
ways to experimental or other empirical writing?" (p.169). 


Bibliometrics—Cross-disciplinary Impact 

Published research is, in essence, a discourse process. Put another way, the rationale for the 
accumulation of a body of scientific literature (archived in academic and professional journals) is to 
impart knowledge to others (the public and students), share findings with colleagues (scholars and 
researchers), and provide a framework for advancing knowledge and (it is hoped) ones discipline. Cited- 
references establish a permanent link in this discourse process. While bibliometrics proper is a subject 
area of study (albeit rather staid) in library and information sciences, citation analysis has captured the 
attention of academics and researchers from all fields and disciplines, particularly in recent years (Bar- 
llan, 2012; Harzing, 2011). Rigorous research on citation analysis has recently appeared in subject areas 
such as allied health (De Groote, 2012), information science (Williams & Winston, 2003), mass media & 
communication (Stephen, 2011), psychology (Cho, Tse, & Neely, 2012; Piotrowski, 2012), social work 
(Hodge, Lacasse, & Benson, 2012), human resource development (Jeung, Yoon, Park, & Jo, 2011), and 
management (Peng & Zhou, 2006; Podsakoff et al., 2008). 

But a legitimate question remains: What's all the fuss about citations? First, and foremost, academicians 
and researchers are interested on the impact of their scholarship and writings. Secondly, science 
requires the interchange of ideas and findings which prompts interest in the "who & what" of scholarly 
influence. Interestingly, recent research shows that in some fields nearly 50% of cited references are less 
than 5 years old (Rethlefsen & Aldrich, 2013). Third, applying metrics to scientific contributions can 
facilitate or identify the structure of science, influential fields of research, and promising avenues of 
discovery (Cacioppo & Cacioppo, 2012). Moreover, vanity extends beyond the individual researcher- 
research institutes and universities have a vested interest in the research ranking of their institutions 
(Egghe, 2010). Most, if not all, disciplines have an inherent need to identify the 'core' journals, e.g., 

Smith (2010), and the top-cited articles (citation 'classics') in their field (Lipsman & Lozano, 2011; Stack, 
2013). Identifying cited-references in books has been a perennial problem and searching options have 
been rather restricted. Recent research, however, has shown that Google Books appears to show 
promise in this endeavor (Kousha & Thelwall, 2009; Kousha, Thelwall, & Rezaie, 2011). In the 
contemporary educational environment, researchers seem keenly interested in the impact of e-journals 
on citation behavior and output (Brown, 2010; Kurtz & Bollen, 2010). Furthermore, advances in 
communication technology and online instruction have prompted investigatory efforts in novel areas 
such as the potential impact of online syllabi on citations (Kousha & Thelwall, 2008) and the impact of 
Twitter on initiation of citation "bursts" (see Eom & Fortunato, 2011; Eysenbach, 2011). 

Research on the topic of citation analysis in the education field has been rather robust, including 
computer science (De Sutter & van den Oord, 2012), health education (Burtis & Taylor, 2010), business 
education (Amara & Landry, 2012; Urbancic, 2011), and STEM education (Greenseid & Lawrenz, 2011). 
Perhaps more pertinent to the current focus, research studies on citation analysis regarding different 
aspects of distance education and e-learning have appeared (Chen & Lien, 2011; Herring, 2010; Hwang 
& Tsai, 2011; Lee et al., 2005; Tuncay & Uzunboylu, 2010). With regard to online instruction, Shih, Feng, 
and Tsai (2008), using a cited-reference strategy, found that studies on instructional approaches, 
information processing, and motivation had the greatest impact on subsequent research. Another study 


reported that editors from distance education journals viewed 'open access' journals quite favorably in 
terms of prestige (Zawacki-Richter et al., 2010). Shaffer (2011), utilizing citation analysis, investigated 
proficiency in graduate students' use of library research skills. 

The Influence of Citation Analysis in Higher Education 

The interface of bibliometrics (defined as: the study and application of indices of scholarship and 
scholarly influence on future research endeavors) and higher education has crystalized in recent years 
largely due to a) the proliferation of scholarly content (entombed mostly in journals and books), and b) 
the availability of cited-reference options across several relatively new database vendors, e.g., Scopus 
and Google Scholar (e.g., van Aalst, 2010). At the same time, there have been major innovations in the 
education field such as the use of computer-mediated technology, online instruction, and open access 
publishing (Hricko & Howell, 2006; Rudestam & Schoenholtz-Read, 2010; Vodanovich & Piotrowski, 
2005). Since the intellectual foundations of education are multidisciplinary, the knowledge base of a 
myriad of fields is the cornerstone that informs and, at the same time, impacts scholarship and 
educational practices (Goodyear et al., 2009; Hart & Metcalfe, 2010). Based on this premise, intellectual 
exchange is a critical function of higher education and academic journals are the primary mode of 
communication and instruction among researchers, professors, and even advanced students. This 
aggregated scholarship functions as a central conduit in the creation, dissemination, and application of 
knowledge. Bibliometrics serves as a barometer of all these major scholarly functions (De Beilis, 2009). 

At the same time, the communicative interaction with regard to scholarly knowledge is rather complex. 
Citation analysis is one critical approach in examining this communicative process and cited references 
are the hallmark in identifying citation patterns (Budd, 1990; Budd & Magnuson, 2010). In fact, citation 
analysis has been regarded as a valid methodology in identifying research trends in the field or in 
examining the rigor, prestige, and impact of individual researchers, journals, and universities (Lee, Wu, & 
Tsai, 2009; Tsay, 2008). Moreover, this bibliometric indicator has been relied upon in evaluating or 
assessing the scholarly influence of individual journals (e.g., Haas et al., 2007; West & Rich, 2012.). 
Scholarly "impact" indices are major criteria in the evaluation of research productivity of academic and 
research faculty, and the critical metric is the number of citations to a faculty member's research 
(Aguinis, Suarez-Gonzalez, Lannelongue, & Joo, 2012). In addition, research studies on citation analysis 
have examined issues such as the international impact of educational research (Vinluan, 2012; 

Wolhuter, 2011) and the use of electronic journals and online reference sources by graduate students 
(Smyth, 2011), and as a valid strategy in detecting plagiarism (Alzahrani, Palade, Salim, & Abraham, 
2012 ). 

One area of intense investigation involves the robust nature of the 'importance' of an individual citation. 
Accordingly, not all citations harbor the same valence or esteem, and the discussion has delineated cited 
reference impact as either 'popularity' (the aggregated number of citations) or 'prestige' (citations 
appearing in highly cited articles or top-ranked journals) (Ding & Cronin, 2011; Zhou, Lu, & Li, 2012). In 
this regard, novel metrics have been introduced to gauge the prestige of both citing and cited journals 
(Guerrero-Bote & Moya-Anegon, 2012). Some researchers have proposed using a 'weighted' 
formulation to determine an article's prestige (Van & Ding, 2010; Zyczkowski, 2010). Others have argued 
that citation indices that purport to ascribe 'prestige' are a misuse of journal impact metrics (Balaban, 


2012). Furthermore, content analyses of the extant literature point to the reality that positive findings 
tend to be cited more than studies reporting negative results (see Fanelli, 2013). 

Despite these academic debates, citation metrics are being used increasingly by educational 
administrators (Rodriguez-Ruiz, 2009). However, as innovative hybrids of the h-lndex (e.g., h int) have 
been introduced in recent years (Guerrero-Bote & Moya-Anegon, 2012; Levene, Fenner, & Bar-llan, 
2012), researchers have expressed concerns about the validity and limitations of these emerging citation 
metrics (Kosmulski, 2010; Schreiber, 2013; Smith, 2012), for faculty evaluation, promotion, or tenure. 
Moreover, educators and academics from within the field of bibliometrics have also scrutinized these 
citation barometers (Chang, McAleer, & Oxley, 2011). Second, the much maligned 'Journal Impact 
Factor' continues to be criticized in the scholarly literature (see Smeyers & Burbules, 2011), highlighting 
issues like the impact of 'self'-citations (Carley, Porter, & Youtie, 2013; Foo, 2009) and editorial board 
influence or pressure on citing their own journal so as to embellish citation counts (Campanario et al., 
2006). A related concern, not frequently mentioned in the published literature, is the issue of reciprocal 
citations within a select, narrow circle of researchers (see Kosmulski, 2010). Interestingly, contrasting 
views on these issues have been noted in the information sciences field, e.g., Krell (2010). 

Critical to the current discussion, researchers are well aware of the fact that educational research 
publications are not well represented among 'source' journals in the major databases that offer cited- 
reference functions and that 'Educational' journals have relatively low impact factor scores (see Togia & 
Tsigilis, 2006). This creates doubt about the salience and usefulness of search files like Web of Science, 
Scopus, and Google Scholar from instructors in the field of education (Albion, 2012). 

Proliferation of Scholarship and Advances in Bibliometrics 

The information age has prompted a dramatic shift in how academic knowledge is created, 
disseminated, communicated, archived, and retrieved (Thelwall, 2009). These advances, however, have 
evidenced a host of concerns with regard to the reliability, credibility, and sustainability of scholarship in 
the published (and online) literature (Meyer & Schroeder, 2009; Narin, Olivastro, & Stevens, 1994; 
Piotrowski & Perdue, 2003). Research studies of online instruction resources have been at the forefront 
of these concerns (e.g., Hricko & Howell, 2006). Over the past decade, bibliometric tools have emerged 
as an efficient and effective remedy for the filtering, evaluation, and accreditation of scholarly material 
(Eom, 2009; Feller, 2005; Richardson & McLeod, 2009; Wolfram, 2003). A decade ago, researchers and 
academicians relied on a sole bibliometric resource, i.e., Social Sciences Citation Index (n.d.), to access 
the scholarly literature in order to access citation reference data on subject matter, target periodicals, 
and author cited references (Herubel & Buchanan, 1993). The past seven years, however, has witnessed 
the proliferation of a number of scholarly citation databases or the introduction of 'citation' options 
within existing database files (see De Beilis, 2009). Moreover, these major citation databases have 
continued to modify and upgrade the operations and functionality of their search platforms at an 
accelerated pace (Harzing, 2011; Moed, 2005; Schroeder, 2007; Taris, 2006). Researchers in the field of 
bibliometrics have also conducted detailed comparisons of citation features in PubMed, Scopus, Web of 
science, and Google Scholar; database functions, including the benefits and shortcomings of each file, 
have been keenly analyzed (e.g., Falagas, Pitsouni, Malietzis, & Pappas, 2008; Garcia-Perez, 2010; Meho 
& Yang, 2007; Vaughan & Shaw, 2008). 


A Note on Open-Access Research 

Open-access has become a well- respected channel for publishing research findings. Recent years have 
witnessed a high level of research interest in examining the influence of open-access journals on the 
visibility level of articles published online and subsequent citation counts (Norris, Oppenheim, & 
Rowland, 2008). Interestingly, most analyses to date indicate that open-access articles have a higher 
probability of being cited than articles limited to traditional toll-access journals. Cross-disciplinary 
studies have confirmed this finding in the field of communication studies (Poor, 2009), biomedicine (Kim 
et al., 2011), anthropology (Xia & Nakanishi, 2012), information sciences (Mukherjee, 2009; Yuan & Hua, 
2011), and sociology (Norris et al., 2008). Researchers have reported that the citation rate 'advantage' 
of open-access articles can be two-fold (see Xia, Myers, & Wilhoite, 2011). In addition, open-access 
have gained momentum in publishing high-quality research and some open access journals are regarded 
as high as their print journal counterparts (Xia, 2012). However, recent bibliometric research suggests 
that more refined investigatory procedures need to be implemented in examining the intricate 
functionality on the process of citing references in the open-access domain (Povh & Zumer, 2012). 

Cross-database Comparisons 

There has been lively and spirited debate on comparative analyses of the attributes and drawbacks of 
the major citation databases (Jasco, 2005; Schroeder, 2007). However, the Web of Science is regarded 
as the 'Gold Standard' or benchmark in bibliometric research (Garcia-Perez, 2011; Harzing, 2013a), 
although the database Scopus provides more comprehensive journal coverage in select fields (Haddow 
& Genini, 2010). Moreover, there has been robust research interest in empirical comparison of 
databases such as Web of Science, Scopus, with Google Scholar (e.g., Falagas et al., 2008; Kousha & 
Thelwall, 2008; Kulkarni, Aziz, Shams, & Busse, 2009; Levine-Clark & Gil, 2009). In one of the few studies 
examining citation-based measurements of 'education' journals, Haddow and Genoni (2009) found that 
Scopus outperformed Web of Science. Perhaps, this finding is a function of the fact that Scopus indexes 
substantially more journals from the education field than Web of Science. 

It must be kept in mind that once a pool of cited references has been identified, a researcher can 
conduct analyses of several key identifiers or factors such as journal or type of journal, language, 
author(s), author affiliation or national residence, topical focus, type of article (empirically-based, 
review, commentary), type of methods or procedures, type of theoretical framework, measures used, or 
trends over time based on publication date, and research funding sources. This exemplifies the robust 
nature of studying citation-based data which is recognized by scholars worldwide (e.g., Aksnes & Rip, 
2009; Bornmann, Schier, Marx, & Hans-Dieter, 2012; Perdue & Piotrowski, 2004). It would be 
informative to provide a brief overview of key studies with a focus on citation analysis across disciplines. 
To that end, Table 1 presents the major bibliometric topics researched in select academic fields; Table 2 
summarizes several key functional aspects of major academic indexes that offer citation-based 
operations and data. 


Table 1 

Examples of Studies on Citation Analysis in Select Disciplines 


Academic Fields 

Study 

Topical Issue 




Accounting 

Chan & Liano (2009) 

Used citation analysis to identify 
top accounting journals and 
institutions: Journal of 

Accounting Research, Journal of 
Accounting and Economics, 
Economics and Accounting 

Review; University of Chicago, 
University of Pennsylvania, 
University of Michigan. 

Biology 

Lortie et al. (2013) 

Critiqued citation analysis in that 
critically important articles in 
evolutionary biology do not 
necessarily receive the highest 
citation rates 

Chemistry 

Bornmann et al. (2009) 

Studied the h-lndex; Journal 

Impact Fact; 20 

organic chemistry journals 

Cognitive Science 

Cho & Neely (2012) 

Examined whether the standard 

H index is superior to variants of 
the index in establishing an 
individual researcher's scholarly 
impact in a sample of elite 
cognitive psychologists 

Communications & Media 

Chung & Park (2012) 

Evaluated Web presence & 
citation counts; 

Scholars in communication 

Computer Science 

Franceschet(2010) 

Highlighted variability in cited- 
references; Web of Science & 
Google Scholar 

Criminal Justice 

Telep (2009) 

Identified the most cited 
research experiments across 6 
criminology areas 

Earth Sciences 

Mikki (2010) 

Compared Google Scholar to 

Web of science 

Economics 

Pislyakov (2009) 

Compared Scopus and Journal 
Citation Reports for leading 
economics journals 

Engineering 

Franceschini & Maisano (2010) 

Commented on h-lndex, h- 
Spectrum; top quality 
engineering-management 
journals 

Environmental Science 

Vanclay (2013) 

Reported that substantive 
'review' articles from high 
impact journals tend to be cited 






most 

Humanities 

Eccles et al. (2012) 

Discussed Webometric analysis 
of digitized scholarly resources 

Human Sexuality 

Antell (2012) 

Studied the extent of coverage 
of LGBT studies in mainstream 

academic literature 

Information Systems 

Cabanac(2012) 

Analyzed editorial board 
involvement across 77 leading 
journals 

Kinesiology 

Omrcen & Lescic (2011) 

Compared citation analysis for 
English vs non-English references 

Knowledge Management 

Bontis & Serenko (2009) 

Studied citation impact of 
intellectual capital journals; h- 
Index; g-lndex 

Management 

Judge et al. (2007) 

Determined that citation rates 
are a function of journal or 
university prestige 

Nursing 

Smith & Hazelton (2008) 

Studied bibliometrics and 

citation-based research in 
nursing profession 

Physical Therapy 

Wiles et al. (2012) 

Evaluated bibliometric patterns 
from 1945-2010 in journal: 
Physical Therapy 

Physics 

Radicchi & Castellano (2011) 

Presented empirical data that 
confirmed that citation analysis, 
over time for a journal, is 
unreliable 

Psychiatry 

Hunt et al. (2010) 

Reported that the h-lndex 
outperformed Journal Impact 
Factor, over time 

Psychology 

Nosek et al. (2010) 

Investigated citation impact & 
career-stage patterns of social 
psychology faculty 


Table 2 

Major Features of Databases that Offer Citation-based Data 



Web of Science 

Google Scholar 

SCOPUS 

PsycINFO 

Coverage 

Extensive, but 
somewhat limited 

in the field of 
education, social 
sciences and 

humanities 

Quite 

comprehensive 
across all fields 

Very 

comprehensive, 
beyond that of 
SSCI/SCI 

Excellent in social- 

behavioral 
sciences; limited 
in management 
and education 





Journal coverage 

15,000+ 

Quite extensive, 
both U.S. and 
foreign 

18,500+ 

About 2,500, but 
article selectivity 
in some journals 

Rapid indexing 

Quite rapid 

Very rapid 

Quite rapid 

2-4 months, but 
can be longer 

Books 

Limited, 

somewhat select 

Moderate, but can 
be selective by 
field 

Limited, 

somewhat select 

Inclusion is 

somewhat 
limited; 
behavioral & 

social sciences 
only 

Conference 

Papers 

Limited coverage 

Moderate 
coverage; select 
Proceedings 

Limited coverage 

No coverage 

Dissertations 

Some coverage 

Extensive 

coverage 

Some coverage 

Inclusion is limited 

to behavioral & 

social sciences 

Graphic metrics 

Available 

Available 

Available 

Not included 


Faculty Naivete Regarding Scholarly Databases 

Survey data indicate that faculty in higher education have a myopic view of the myriad of scholarly 
database selections currently available (Piotrowski, Perdue, & Armstrong 2005). These authors found 
that instructors tend to focus on a limited number of online databases when conducting research. 
Unfortunately, this relative lack of orientation or competency (familiarity?) on the part of faculty 
regarding the nuances of scholarly database searching, let alone the complexities of citation searching, 
is unwittingly transferred on to students. For example, there has been abject neglect in the social 
sciences to remedy this state of affairs (see Piotrowski, 2007 for a discussion on this issue). This 
pedagogic neglect of online searching methods has continued over the past decade (Dron, 2012; 
Vodanovich & Piotrowski, 2001), and evidenced by recent national survey data (see Piotrowski & 
Vodanovich, 2004). Fortunately, investigators are now focused on the role of the student and the 
process of engaging in research projects. In fact, recent research has examined the comfort level of 
graduate students' use of Google Scholar as a research retrieval tool (Cothran, 2011). 

More research on the acquisition of competency in bibliometric research methods for the novice learner 
is sorely needed, considering the evolving and intricate nature of the field (e.g., Bornmann et al., 2011; 
Franceschet, 2010; Kousha & Thelwall, 2007). There is an urgent need to discuss the fundamental 
operations and functions central to citation searching across select database vendors, particularly for 
faculty facing academic milestones like obtaining tenure, promotion, and applying for research grants. 

Conclusions 

If a scholar wants to identify a listing of their cited author references contained in books or book 
chapters, then Google Scholar-Books would seem in order. Also, in the field of Psychology, PsycNet 
offers a 'cited by' option that includes books/chapters, although coverage is somewhat limited (Perdue 












& Piotrowski, 2003). When faculty need to document their citations found in peer-reviewed articles, 
then Web of Science and Scopus have comprehensive coverage of scholarly and academic journals (but 
limited coverage of books, dissertations, or proceedings). However, authors, in need of locating citations 
in 'open access' journals, appear to be limited to searching Google Scholar (Aguillo, 2012; Herther, 2010; 
Varshney, 2012). 

Recently, Carleton, Parkerson, and Horswill (2012) concluded that all citation databases demonstrate 
restrictions in function and capacity, and suggest that a multi-database search strategy should provide 
comprehensive, optimal results. This approach is supported by prior research findings on the utility of 
multi-file search strategies when conducting research (Lohonen, Isohanni, Nieminen, & Miettunen, 

2010; Piotrowski & Perdue, 1988; Wu, Aylward, Roberts, & Evans, 2012). Based on this review of the 
extant literature, "Online" educators have a functional pedagogic framework to a) utilize available 
citation-based features of established databases, and b) consider the limitations of more recent 
bibliometric tools in the field of citation analysis. With 'hands-on' experience and attention to 
developments in this rapidly advancing field, educators can stay abreast of the most efficacious methods 
needed for obtaining optimum citation analysis results across academic disciplines. 
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